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1.0  SUMMARY 


The  Air  Force  regards  Spatial  Disorientation  (SD)  and  loss  of  situational  awareness  (SA) 
as  contributing  factors  in  operational  Class  A  ($1M  in  aircraft  loss  and/or  pilot  fatality) 
aircraft  mishaps.  Spatial  Disorientation  (SD)  is  defined  as  an  incorrect  sense  in-flight  of 
one’s  position,  motion,  or  attitude  with  respect  to  the  earth’s  surface  (14).  Situational 
awareness  is  the  perception  of  the  elements  in  the  environment  within  a  volume  of  time 
and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the 
near  future  (14).  Multisensory  systems  have  been  developed  to  enhance  the  aircraft 
attitude  information  to  the  pilot.  The  systems  incorporate  multisensory  aids  including 
helmet-mounted  symbology  and  tactile  and  audio  cues.  Two  systems  have  been 
prototyped  and  demonstrated  in  the  Air  Force  Research  Laboratory.  The  technology 
could  have  application  in  the  rotary-wing  brownout  program.  This  paper  discusses  the 
development  of  two  systems  and  a  potential  application  including  an  augmented 
cognition  application.  Unlike  automatic  ground  collision  avoidance  systems,  these 
systems  do  not  take  over  the  aircraft  if  a  pre-set  altitude  is  broached  by  the  pilot;  rather, 
these  multisensory  systems  provide  complementary  attitude  cues  to  the  pilot  via  the 
tactile,  audio,  and  visual  systems  that  allow  the  pilot  to  continue  flying  through 
disorienting  conditions. 

2.0  INTRODUCTION 

Current  attitude  information  is  presented  to  the  pilot  by  the  head-up  display  (HUD)  and 
head-down  displays  (HDD).  The  primary  flight  instruments  (HDD)  have  changed  very 
little  over  the  past  75  years.  Pilots  are  still  trained  to  “get  onto  the  round  dials”  when 
they  are  disoriented  or  looking  for  reliable  aircraft  attitude  information  in  flight.  The 
standard  head-down  displays  include  the  altimeter,  airspeed  indicator,  heading  indicator, 
and  attitude  display,  which  are  arranged  in  a  “T”  formation.  This  arrangement  allows  the 
pilot  to  quickly  scan  these  instruments  and  determine  altitude,  airspeed,  heading,  and 
aircraft  attitude  (pitch/bank).  The  MIL-STD  HUD  is  characterized  by  MIL-STD-1787C 
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(11).  In  the  HUD,  a  pitch  ladder  symbology  set  is  provided  that  moves  up  and  down  with 
pitch  and  bank  of  the  aircraft.  The  HUD  is  a  fixed,  see-through  combiner  glass  mounted 
in  the  cockpit  on  the  glare  shield  just  above  the  HDD.  Symbology  is  displayed  as  a 
virtual  image  on  the  glass,  which  allows  the  pilot  to  read  the  flight  reference  information 
with  his/her  eyes  focused  at  infinity  (>  20  feet). 

Helmet-mounted  displays  (HMDs)  are  becoming  more  operationally  capable  and 
integrated  into  more  and  more  fixed-wing  aircraft  such  as  the  F/A-18  and  the  F-35  Joint 
Strike  Fighter  (JSF).  The  JSF  program  office  is  looking  into  a  replacement  for  the  hard- 
mounted  HUD  in  the  cockpit.  MIL-STD-1787C  symbology  will  be  displayed  on  the 
visor  of  the  JSF  helmet.  The  challenge  with  HMDs  is  to  provide  information  in  the  HMD 
that  is  compatible  with  the  HDD.  This  information  typically  includes  altitude,  attitude, 
heading  and  pitch/bank  information.  One  of  the  complaints  about  putting  the  current 
HUD  information  in  the  HMD  is  the  excessive  clutter  of  the  climb-dive  ladder  with 
horizontal  lines  pitching  and  rolling  throughout  the  limited  field-of-view  (FOV).  HMD 
developers  have  devised  an  array  of  alternate  attitude  displays  for  the  off-boresight 
attitude  reference.  When  coupled  with  a  head-tracking  system,  the  HMD  can  display 
more  detail-rich  symbology  on-boresight  and  less  symbology  when  viewed  off-boresight. 
The  challenge  to  the  HMD  symbology  developer  is  to  develop  both  bank  and  climb-dive 
information  in  one  display  minus  the  disconcerting  ladder  lines  streaming  through  the 
pilot’s  FOV.  One  concept  in  the  F-35  is  to  display  the  MIL-STD-1787C  HUD 
information  on-boresight  in  the  HMD,  but  to  have  it  drop  out  of  the  display  once  the  pilot 


looks  off-boresight  (>30  degs). 


2.1  Spatial  Orientation  Retention  Device  (SORD) 

The  Spatial  Orientation  Retention  Device,  or  SORD,  is  a  multisensory  cueing  system  that 
includes  tactile,  audio,  and  visual  components  that  are  integrated  with  a  smart  model  of 
human  orientation  (Figure  1).  The  concept  of  SORD  is  to  provide  additional 
multisensory  information  to  the  pilot  about  his  or  her  aircraft  attitude  (1).  Many  SD 
mishaps  can  be  attributed  to  pilot  distraction  or  channelized  attention  in  flight,  during 
which  the  aircraft  slips  into  an  unusual  attitude  so  gradually  as  to  remain  undetected  by 
the  pilot’s  vestibular  or  somatosensory  systems  (14).  In  many  instances,  the  aircraft 
crashes  before  the  pilot  realizes  the  unusual  attitude.  SORD  provides  tactile,  audio,  and 
visual  information  to  the  pilot  in  addition  to  the  usual  HDD  or  HUD  attitude  information 
already  available.  SORD  is  an  alerting  system  for  the  pilot. 

SORD  has  four  components:  tactile  garment,  3-D  audio,  HMD  symbology,  and  a  Spatial 
Orientation  (SO)  assessor  (Figure  1).  The  tactile  garment,  3-D  audio,  and  HMD 
symbology  have  been  demonstrated  in  the  Air  Force  Research  Laboratory  (AFRL).  A 
prototype  of  a  tactile  cueing  system  incorporating  electric  tactors  has  also  been  evaluated. 
The  tactile  cueing  system  that  has  been  evaluated  is  a  vibro-mechanical  tactor  system. 

The  3-D  audio  system  demonstrated  is  the  NASA  SLAB  system  which  modulates  the 
volume  of  white  noise  in  either  the  left  or  right  ear-cup  depending  on  the  direction  of 
bank  or  roll  of  the  aircraft  (13).  The  off-boresight  arc  segment  attitude  reference  (ASAR) 
symbology,  also  called  the  Non-Distributed  Flight  Reference  (NDFR),  allows  the  pilot  to 
view  attitude  information  off-boresight  when  the  symbology  is  displayed  on  an  HMD. 

The  assessor  is  a  mathematical  model  of  human  spatial  orientation  that  runs  in  real-time 
in  a  microcomputer. 
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Figure  1.  Traditional  aircraft  pilot-control  loop  with  SORD  shown  in  dashed  area 
2.2  Tactile  Cueing 

Tactile  cueing  is  not  new  technology  (5,  6,  15).  It  has  been  demonstrated  extensively  in 
the  Tactile  Situation  Awareness  System  (TSAS)  developed  by  the  Naval  Aeromedical 
Research  Laboratory  (NAMRL)  and  at  TNO  in  the  Netherlands  (5,  6).  Tactile  torso 
displays  consist  of  numerous  vibrators  or  tactors  that  are  in  a  matrix  arrangement  and 
have  contact  with  the  chest,  back,  and  sides  of  the  wearer.  The  tactile  suits  range  in 
number  of  tactors  from  8  to  64.  The  concept  is  to  apply  a  mechanical  or  electrical  signal 
to  the  skin  of  the  wearer  via  this  matrix  of  tactors  in  order  to  convey  a  signal  or  series  of 
signals  that  have  meaning,  much  like  one  taps  a  person  on  the  shoulder  in  order  to  gain 
their  attention.  The  tactors  confer  information  to  the  wearer  by  presenting  localized 
vibrations  or  electrical  signals.  By  energizing  a  row  of  tactors,  the  horizon  line  can  be 
emulated,  completely  encircling  the  wearer.  By  powering  just  one  or  several  of  the 
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tactors,  the  pilot/wearer  can  be  alerted  to  an  over-banked  situation  such  as  a  30-deg  roll  to 
the  left  (Figure  2).  The  tactile  cues  that  indicate  over-banking  of  the  simulated  aircraft 
have  been  evaluated  during  emulated  visual-vestibular  illusions  on  the  Dynamic 
Environment  Simulator  (DES)  centrifuge,  a  dynamic  flight  simulator  at  AFRL. 


Figure  2.  Localized  tactile  cue  indicating  a  30-degree  bank  to  the  left 
2.3  3-D  Audio 

3-D  audio  is  not  a  new  technology.  It  has  been  demonstrated  in  the  laboratory  for  20 
years  and  has  been  considered  for  operational  aircraft  (9,  10).  The  NASA  SLAB  is  the 
audio  cueing  system  implemented  for  the  demonstration  of  SORD.  SLAB  is  a  real-time 
virtual  acoustic  environment  rendering  system  that  is  available  from  the  Spatial  Auditory 
Displays  Lab  at  NASA  Ames  Research  Center  (12,  13).  SLAB  performs  spatial  3-D 
sound  processing,  allowing  the  arbitrary  placement  of  sound  sources  in  auditory  space. 
SLAB  is  integrated  with  a  Polhemus  (Colchester  VT)  head  tracker  that  provides  head 
orientation  information  to  the  SLAB  software.  SLAB  is  currently  being  used  with  a 
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white  noise  source  that  increases  in  volume  as  the  subject  rolls  left  (left-ear  volume 
increases)  or  right  (right-ear  volume  increases). 

2.4  ASAR  (NDFR) 

The  ASAR  represents  the  visual  display  symbology  that  is  being  evaluated  under  the 
SORD  program.  In  1987  Domier  developed  the  ASAR  as  an  alternate  flight  reference 
symbology  to  the  standard  HUD  climb-dive  ladder  (3).  The  ASAR  or  “Orange  Peel” 
display,  as  it  has  been  called,  has  been  demonstrated  to  accurately  convey  aircraft  attitude 
information  to  the  pilot  without  displaying  the  climb-dive  ladder.  The  ASAR  conveys 
dive  angles  by  increasing  the  amount  of  attitude  arc  displayed  to  form  a  more  complete 
circle.  During  climb,  the  attitude  arc  shrinks  to  indicate  increasing  climb  angles  until,  at 
90  degrees  climb,  only  the  ASAR  gap  marks  remain  (Figure  3).  Bank  angles  are 
determined  by  comparing  the  center  of  the  attitude  arc  to  the  aircraft  symbol.  The  ASAR 
was  modified  to  employ  dots  and  gaps  at  +/-  30  degrees  and  +/-  60  degrees  of  climb-dive; 
the  modified  display  is  referred  to  as  the  Non-Distributed  Flight  Reference  (NDFR)  (7, 

8).  The  advantages  offered  by  the  NDFR  include  its  compact  size  and  the  ability  to 
display  it  on  an  HMD  off-boresight. 


Figure  3.  Advanced  Non-Distributed  Flight  Reference  (NDFR) 
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2.5  Smart  Orientation  Processor 


The  smart  processor  is  the  “brains”  behind  the  multisensory  cueing  system.  The 
Disorientation  Analysis  and  Prediction  System  takes  actual  flight  information  (linear  and 
rotational  velocities  and  accelerations,  aircraft  attitude  information,  and  airspeed/altitude) 
and  processes  these  data  through  a  visual-vestibular-tactile  mathematical  model  of  the 
human  to  develop  an  orientation  prediction  for  the  human  (2).  This  prediction  of 
orientation  is  compared  to  the  actual  aircraft  orientation,  and  a  difference  between  the 
model-predicted  orientation  and  the  actual  orientation  is  developed.  This  difference  is 
termed  the  “disorientation  index”  in  the  model. 

2.6  Fixed  Wing  Applications 

SORD  has  been  demonstrated  without  the  smart  processor  in  the  laboratory  and  in  the 
DES  centrifuge.  The  prototype  SORD  provides  multisensory  information  via  tactile  cues 
on  the  subject’s  sides  (over  30  degrees  bank  either  right  or  left),  auditory  cues  via  the 
headset  (volume  changes  as  the  simulated  aircraft  is  banked),  and  the  ASAR  display 
which  is  displayed  via  an  HMD  that  provides  both  a  virtual  image  as  well  as  a  see- 
through  capability.  The  ASAR  provides  aircraft  attitude  information  (it  displays  what  the 
nose  of  the  aircraft  is  doing  as  one  looks  off-boresight)  by  interpreting  the  arc  segment, 
its  orientation,  and  its  shape.  The  ASAR  allows  the  subject  to  look  for  out-the-cockpit 
targets  without  having  to  look  back  inside  the  cockpit  either  to  perform  the  “T 
crosscheck”  or  to  interpret  the  hard-mounted  HUD.  The  ASAR  provides  altitude, 
airspeed,  pitch/bank  and  heading  information  all  in  one  condensed  symbol  set  (Figure  3). 


When  integrated  with  a  smart  processor  (illustrated  as  the  spatial  orientation  (SO)  model 
assessor  in  Figure  1),  SORD  could  operate  in  an  active  or  “idle”  state.  If  the  pilot  wanted 
SORD  to  be  inactive,  he/she  would  select  SORD  to  be  inactivated.  When  active,  the 
assessor  would  begin  comparing  actual  flight  data  with  a  computed  spatial  orientation 
model  of  the  human.  If  the  aircraft  were  banked  more  than  a  predetermined  set  point 
(e-g.,  >  30  degrees),  the  tactile  cues  would  be  activated  (vibration  on  the  side  of  the  pilot), 
the  3-D  audio  would  indicate  a  greater  volume  in  one  ear-cup  vs.  the  other,  and  the 
ASAR  symbology  would  indicate  an  over-banked  condition.  The  smart  assessor  could 
determine  when  disorienting  conditions  are  present  in  the  cockpit  and  would  implement 
the  cueing  strategy  (tactile  +  audio,  ASAR  only,  audio  only,  etc).  The  goal  is  to  provide 
SORD  technology  for  all  military,  commercial  and  general  aviation  cockpits. 

2.7  Spatial  Orientation  Enhancement  System  (SOES) 

The  Spatial  Orientation  Enhancement  System  (SOES)  was  developed  by  Rockwell- 
Collins  in  conjunction  with  the  Operator  Performance  Lab  at  the  University  of  Iowa. 
Under  a  Dual  Use  Science  &  Technology  effort  with  AFRL,  the  SOES  was  developed 
and  evaluated  at  the  Dynamic  Environment  Simulator  centrifuge  (AFRL/HEPG)  as  well 
as  in  flight  in  Canada  (16).  Only  the  DES  evaluation  is  discussed  here. 

SOES  included  a  tactile  suit  (Figure  4),  an  audio  (sonification)  system  to  provide 
orientation  cues  and  a  synthetic  vision  system  to  provide  synthetic  terrain  when  there  was 
no  out-the-window  visibility.  The  contribution  of  these  orientation  aids  was  evaluated 
individually  and  in  combination  as  subjects  were  presented  with  unusual  attitudes  in  the 
DES  and  subjected  to  vestibular  illusions. 
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Figure  4.  SOES  tactile  suit  system 
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Figure  5.  Subject  with  SOES 


3.0  Methods 

Twelve  military  pilots  participated  in  the  study.  Eight  were  Visual  Flight  Rules  qualified. 
After  receiving  a  briefing  and  fixed-base  simulator  training,  the  subjects  received  training 
in  the  DES.  A  subject  fitted  with  SOES  is  shown  in  Figure  5.  The  dependent  variables 
were  sonification  (with/without),  tactile  cues  (with/without),  background  attitude 
indicator  (with/without),  and  synthetic  vision  display  (with/without).  Subjects  were 
presented  with  a  somatogravic  illusion  (pitch  up  and  pitch  down)  and  a  leans  illusion  (left 
and  right).  Subjects  experienced  1 2  exposures  each.  SOES  was  evaluated  subjectively 


on  a  scale  of  0  =  low  to  10  =  high.  Subjective  ratings  of  the  systems  were  tabulated 
(Figure  6). 


Information  Source 


Figure  6.  Subjective  results  of  SOES  in  DES 


4.0  Results  and  Discussion 

Subjects  rated  the  BAI,  background  attitude  indicator  (7.9)  and  HDD,  head-down  display 
(7.9)  higher  than  the  other  displays.  Sonification  was  rated  the  lowest  (5.6),  followed  by 
the  tactile  suit  (6.7)  and  the  AI,  attitude  indicator  (7.2).  The  DES  was  considered  an 
excellent  device  for  studying  spatial  orientation  performance  in  flight  environments  by 
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the  researchers.  Tactile  and  audio  cues  did  not  appear  to  be  pre-cognitive  without 
prolonged  exposure  and  over-learning  since  some  effort  was  required  to  “decode”  the 
meaning  of  the  tactors  and  sonification  cues.  The  tactile  cues  did  help  reduce  aileron 
reversals  during  recoveries  from  extreme  bank  angles;  tactile  cues  also  resulted  in  smaller 
pitch  excursions  and  much  smaller  altitude  gain/loss  during  recovery  from  extreme  pitch 
angles.  The  sonification  cues  caused  higher  subjective  workload  in  decoding  and  larger 
pitch  excursions  during  recovery  from  extreme  pitch  angles. 

4.1  Rotary  Wing  Applications  of  Orientation  Enhancement  Systems 
SORD/SOES  technology  is  also  applicable  to  the  helicopter  flight  environment.  The 
tactile  and  audio  cues  can  provide  information  to  the  pilot  about  helicopter  attitude,  crew 
resources,  and  situational  awareness.  An  NDFR-like  display  on  a  helmet  visor  would  be 
valuable  to  the  helicopter  pilot  as  it  would  provide  critical  helicopter  attitude  information 
as  the  pilot  looks  off-boresight.  The  3-D  audio  cues  would  have  two  benefits:  1)  the 
voice  communications  of  ground  control,  the  co-pilot,  flight  engineer  and  other  crew  in 
the  rear  of  the  helicopter  can  be  spatially  separated,  allowing  the  pilot  to  hear  the 
communication  from  the  actual  direction  of  the  crewmember,  thus  reducing  the  pilot’s 
workload;  and  2)  the  communications  from  other  helicopters  can  be  tracked  and 
presented  to  the  pilot  from  their  respective  locations.  If  there  is  a  multiple  helicopter 
landing,  the  lead  helicopter  can  communicate  with  the  other  helicopters  and  locate  them 
audibly  and  spatially  with  respect  to  the  lead’s  location.  This  would  increase  situational 
awareness  during  brownout  landings  when  the  helicopters  cannot  see  one  another. 

Tactile  cues  delivered  via  the  seat  pan  would  provide  obstacle  avoidance  warning  to  the 
pilot. 
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4.2  Functional  Brain  Monitoring  (ASOS) 

The  Augmented  Spatial  Orientation  System  (ASOS)  is  SORD  or  SOES  with  functional 
brain  monitoring  (electroencephalogram).  ASOS  will  include  a  dry  electrode  ensemble 
that  will  be  incorporated  into  the  flight  helmet.  The  electrodes  will  monitor  the 
frequency  of  the  EEG  and  a  computer  program  will  detect  when  the  EEG  is  indicating  an 
“overload”  situation  in  the  pilot.  The  EEG  system  will  also  be  programmed  to  detect 
when  Type  II  (recognized)  illusions  are  detected.  Gallimore  et  al.  found  significant  EEG 
responses  for  two  in-flight  illusions  during  a  simulator  study  involving  Air  Force  pilots  in 
a  ground-based  trainer  (4).  The  functional  state  of  the  pilot  will  be  monitored  via  the 
electroencephalogram.  If  the  pilot  is  determined  to  be  task  loaded,  or  a  pattern  of 
electrode  voltage  signals  a  particular  illusion,  the  multisensory  cues  to  the  pilot  can  be 
activated.  If  the  aircraft  were  banked  more  than  a  predetermined  set  point  (e.g.,  >  30 
degrees),  the  tactile  cues  could  be  activated  (vibration  on  the  side  of  the  pilot),  the  3-D 
audio  would  indicate  a  greater  volume  in  one  ear-cup  vs.  the  other,  and  the  ASAR 
symbology  would  indicate  an  over-banked  condition.  The  smart  assessor  could 
determine  when  disorienting  conditions  are  present  in  the  cockpit  in  conjunction  with  the 
brain  monitoring  system  and  would  implement  the  cueing  strategy  (tactile  +  audio, 

ASAR  only,  audio  only,  etc). 

5.0  Conclusion 

A  multisensory  aircraft  attitude  tool  is  described  that  will  allow  the  pilot  to  channelize 
his/her  attention  and  spend  more  time  on  out-the-cockpit  visual  tasks  without  having  to 
continuously  bring  his/her  vision  back  into  the  cockpit  to  monitor  aircraft  attitude 
instruments.  Aircraft  attitude  information  currently  displayed  on  HDD  and  HUD  is 
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supplemented  by  tactile  cues,  audio  cues,  helmet-mounted  symbology  or  synthetic  vision 
displays  that  reinforce  attitude  information  about  the  state  of  the  aircraft  in  real  time. 
Tactile  and  audio  cues  give  the  pilot  information  about  the  airspeed,  altitude,  heading, 
and  bank  and  pitch  of  the  aircraft  without  having  to  constantly  monitor  in-cockpit  HUD 
and  HDD  displays.  The  addition  of  functional  brain  monitoring  will  provide  augmented 
cognition  and  help  determine  when  the  pilot  is  task  loaded  or  experiencing  disorientation. 
SORD  and  SOES  can  help  reduce  pilot  workload  by  reassuring  the  pilot  about  his/her 
aircraft  attitude  and  eliminating  the  requirement  of  having  to  focus  on  the  cockpit 
displays.  Potential  application  of  the  technology  to  the  low/no  visibility  brownout 
problem  is  also  under  evaluation. 
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